Confidence-Based Assessment of Two-Alternative Format Tests

The effect of guessing in multiple choice tests is usually reduced by using a large number of test items and increasing the number of choices per item. The paper addresses the important and practical issue of grading Two-Alternative format tests, that is, decision questions of the true/false type, where the answer is only one of two choices .This format is not only easy to construct, but could reflect real life situations such as when a judge has to decide between a GUILTY or a NOT GUILTY verdict, or when a computer science student is asked whether a given computational problem is NP-hard or not. In binary-choice tests where the number of choices per test item is restricted to two, the degree of certainty that the student has in answering the test item becomes an important factor in assessing his/her score. The standard procedure for incorporating certainty/confidence levels in grading multiple-choice tests is to have each student attach one of several, usually four, confidence levels to each question separately. In this paper we suggest a different confidence-based procedure for grading binary-choice tests. Assuming that humans tend to be more comfortable in assessing their confidence level to different things in a relative manner rather than in an absolute manner, the concept of relative uncertainty is used. Each student is asked to rank the test items relative to each other according to his/her confidence level in answering the item correctly. The test is graded according to the rank sequence produced. There is no penalty for wrong answers. A step-size is used to determine changes in the student’s confidence level based on the number of incorrect answers made. A reduction function is used to determine the scores to correct answers at the different confidence levels. By varying the step-size and the reduction function, many different grading strategies can be obtained.